Identifying expertise and using it to extract the Wisdom of the Crowds

The "wisdom of the crowds" refers to the ability of statistical aggregates based on multiple opinions to outperform individuals, including experts, in various prediction and estimation tasks. For example, crowds have been shown to be effective at forecasting outcomes of future events. We seek to improve the quality of such aggregates by eliminating poor performing individuals, if we can identify them. We propose a new measure of contribution to assess the judges' performance relative to the group and use positive contributors to build a weighting model for aggregating forecasts. In Study 1, a dataset of 1,233 judges forecasting 104 binary current events served to illustrate the superiority of our model over unweighted models and models weighted by measures of absolute performance. We also demonstrate the validity of our model by substituting the judges with fictitiously "random" judges. In Study 2, we used a dataset of 93 judges predicting winners of games for the 2007 NFL season and evaluated our model’s rate of convergence with a benchmark of simulated experts. The model does an excellent job of identifying the experts quite quickly. We show that the model derives its power by identifying experts who have consistently outperformed the crowd.